Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try pnpm (take two) #38624

Closed
wants to merge 27 commits into from
Closed

Try pnpm (take two) #38624

wants to merge 27 commits into from

Conversation

sarayourfriend
Copy link
Contributor

Description

I'll force push this onto try/pnpm in #37324 once this is working with CI so we can preserve the original PR discussion.

Testing Instructions

Screenshots

Types of changes

Checklist:

  • My code is tested.
  • My code follows the WordPress code style.
  • My code follows the accessibility standards.
  • I've tested my changes with keyboard and screen readers.
  • My code has proper inline documentation.
  • I've included developer documentation if appropriate.
  • I've updated all React Native files affected by any refactorings/renamings in this PR (please manually search all *.native.js files for terms that need renaming or removal).
  • I've updated related schemas if appropriate.

@sarayourfriend sarayourfriend force-pushed the try/pnpm-fresh branch 2 times, most recently from 9de3efb to 96edaf8 Compare February 8, 2022 13:55
@sarayourfriend
Copy link
Contributor Author

The Pull request automation job is going to continue failing until the new action I've added is on trunk. Not sure there's a way around that for this initial PR.

@sarayourfriend
Copy link
Contributor Author

Failing on installation due to some patches not applying correctly. Either we need to hoist the packages we're patching or I think we could also just change the patch diffs to point to the .pnpm prefix that pnpm uses. I'll experiment with both, first trying to reproduce the issue locally.

@sarayourfriend
Copy link
Contributor Author

I think I need to bust the cache somehow 🤔 Seems like the cache calculation should include .npmrc too 😅

@sarayourfriend
Copy link
Contributor Author

sarayourfriend commented Feb 8, 2022

That worked!

Now there seem to be some TypeScript resolution issues. Cursory searching seems to indicate TypeScript should "just work" with pnpm but clearly there's some issue with our setup here.

I also can't reproduce the issue locally yet, so that's probably the first step.

Will report back anything I find 🙂

@sarayourfriend
Copy link
Contributor Author

Aha! Looking at the first couple types that aren't being resolved it seems like they're indeclared development dependencies! pnpm is already doing it's job for us and pointing out assumptions we've made in the dependency chain! Marvelous 🤩

@sarayourfriend sarayourfriend force-pushed the try/pnpm-fresh branch 2 times, most recently from 9858629 to ee56705 Compare February 8, 2022 14:53
@sarayourfriend
Copy link
Contributor Author

Had to squash changes to make rebasing easier when package-lock.json changes on trunk. Sorry if this makes the history and comments confusing!

@sarayourfriend
Copy link
Contributor Author

The plugin-zip building script needs to be updated to use pnpm. npm cache verify doesn't need to happen anymore I don't believe as pnpm will automatically verify the cache when running pnpm install (maybe??). I'll do some more research on this and try to discover what the right way of doing that with pnpm is.

@sarayourfriend
Copy link
Contributor Author

This will on some level be blocked by until this PR is merged for the compressed-size-check action: preactjs/compressed-size-action#68

@sarayourfriend
Copy link
Contributor Author

Using pnpm to run the build script seems to have moved the progress on this PR backwards 😰 I need to put this down for now but will revisit it next week again (when there are more merge conflicts to resolve, I am sure!)

@sarayourfriend sarayourfriend mentioned this pull request Feb 8, 2022
Copy link
Member

@kevin940726 kevin940726 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for starting this! Spotted some tiny errors.

Will be happy to try it out once this is ready 👍

with:
node-version: ${{ matrix.node }}
cache: npm
node-version: ${{ matrix-node }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should still be matrix.node?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops, yes it should be!

with:
node-version: 14
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to keep this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default is set to 14. I only kept passing it to the action in places that were still using the matrix strategy for it.

- name: Install
if: inputs.install
shell: bash
run: "pnpm install ${{ inputs.frozen-lockfile && '--frozen-lockfile' || '' }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the doc, seems like we don't have to pass --frozen-lockfile to true in CI since it's the default.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. I don't really understand the "For CI: true" remark. How can it tell if it's in a CI environment? This isn't described anywhere in the documentation as far as I could tell... might have to dig into the code to find that out when I come back to this.

I do think you're right in questioning why we would need it to ever not install with a frozen-lockfile on CI though. Maybe this PR is the right time to investigate that? 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe most programs depend on the env variable process.env.CI to exist in CI environment. Most CI platforms, including GH Actions, set this variable.

required: false
description: Optional working directory to run installation in.
frozen-lockfile:
default: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we want to default it to false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure! There were places that were doing just npm install rather than npm ci and I didn't want to change their behavior in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In bd1d499 I changed it to just do a regular frozen-lockfile install everywhere relying on the default behavior in CI you mentioned. I can't think of a reason why CI should need to use a non-frozen install anywhere anyway.

default: false
description: Whether to install dependencies using `--frozen-lockfile`.
node-version:
default: 14
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is nice that we don't have to repeat passing node-version in every workflow if it's just running the default version. We can just update this file when upgrading node to a major version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this could also be changed to node-version-file: '.nvmrc' so that https://github.com/WordPress/gutenberg/blob/trunk/.nvmrc is the only place to change the version.

https://github.com/actions/setup-node/blob/main/docs/advanced-usage.md#node-version-file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find! That sounds like a good option 🎉 We'd need to keep around node-version still for the workflows that use a node version matrix but referring to the .nvmrc file for the default is 💯

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.nvmrc plays an important role for branches where we apply fixes to older WordPress major releases like wp/5.1.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay in bd1d499 I tried to set it up to just use .nvmrc by default unless a specific node-version is passed in (to preserve the ability for workflows to use the node version matrix, if that's still necessary). Hopefully it works, I'm not too strong on GitHub action expressions but should only need a little tweaking if it doesn't work right off the bat.

package.json Outdated Show resolved Hide resolved
packages/scripts/scripts/check-licenses.js Outdated Show resolved Hide resolved
@gziolo gziolo added [Type] Build Tooling Issues or PRs related to build tooling Gutenberg Plugin Issues or PRs related to Gutenberg Plugin management related efforts labels Feb 12, 2022
@sarayourfriend
Copy link
Contributor Author

I just realized I'd been fighting dependency resolution issues through hoisting that are meant to be resolved through cross-workspace-package dependencies... Turns out file is meant to be replaced with workspace.

@sarayourfriend
Copy link
Contributor Author

sarayourfriend commented Feb 12, 2022

Turns out eslint and stylelint are both very special and refuse to resolve plugin modules in a modern way. eslint has a widely used patch for this behavior but stylelint does not, so the latter requires us to hoist its dependencies.

It turns out the patch will only work for a top-level eslint configuration. The nested react-native-editor configuration won't be helped by the patch and requires hoisting of the eslint-plugin dependencies anyway. I removed the patch and added a lengthy explanation for why we're hoisting those packages.

In general I think it'd be a good idea to continue documenting hoists. I feel like we can avoid just hoisting everything (which has it's own set of problems anyway) and everything that is documented and hoisted should theoretically be able to eventually be removed with the right dependency declarations.

Of course, that's with the exception of stylelint, eslint and anything used by expo, because those tools all have their own non-modern module resolution strategies that cause problems with pnpms strictness.

Consumers of the wp-scripts package outside the monorepo will not be effected by this as the dependencies will install at the top level of their repositories (unless they're also using a monorepo, I suppose, but then only if they use pnpm will they run into it).

Anyway, it's kind of a mess but unavoidable at this point as far as I can tell.

@sarayourfriend
Copy link
Contributor Author

sarayourfriend commented Feb 13, 2022

Regarding the current state of the unit tests: I think they might be flagging a potentially problematic (not generally, just for pnpm's strictness) assumption in the WebpackExtractDependenciesPlugin.

The fact that the e2e tests are passing consistently is exciting though!

@sarayourfriend
Copy link
Contributor Author

Closing this draft. If anyone wants to pick this up once React Native is supported by pnpm then you'll have some starting points here though I would be things have changed in the mean time. The new npm versions might not make this change worthwhile anyway.

@gziolo gziolo deleted the try/pnpm-fresh branch September 20, 2022 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Gutenberg Plugin Issues or PRs related to Gutenberg Plugin management related efforts [Type] Build Tooling Issues or PRs related to build tooling
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants